Generating Holistic 3D Scene Abstractions for Text-based Image Retrieval

摘要

Spatial relationships between objects provide important information fortext-based image retrieval. As users are more likely to describe a scene from areal world perspective, using 3D spatial relationships rather than 2Drelationships that assume a particular viewing direction, one of the mainchallenges is to infer the 3D structure that bridges images with users' textdescriptions. However, direct inference of 3D structure from images requireslearning from large scale annotated data. Since interactions between objectscan be reduced to a limited set of atomic spatial relations in 3D, we study thepossibility of inferring 3D structure from a text description rather than animage, applying physical relation models to synthesize holistic 3D abstractobject layouts satisfying the spatial constraints present in a textualdescription. We present a generic framework for retrieving images from atextual description of a scene by matching images with these generated abstractobject layouts. Images are ranked by matching object detection outputs(bounding boxes) to 2D layout candidates (also represented by bounding boxes)which are obtained by projecting the 3D scenes with sampled camera directions.We validate our approach using public indoor scene datasets and show that ourmethod outperforms baselines built upon object occurrence histograms andlearned 2D pairwise relations.